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DETAILED ACTION 
Response to Amendment 

1. Applicant's amendment filed on April 14, 2005, has been entered in full. 

2. Applicant's arguments on pages 6-10 in amendment filed April 14, 2005, with respect 
to the rejection(s)of claim(s) 1-13 under 35 U.S.C. 103(a) respectively, have been fully 
considered but are not persuasive. See the detailed discussion in section entided "Response 
to Arguments". 

Claim Rejections - 35 USC§ 103 

3. The following is a quotation of 35 U.S.C. 103(a) which forms the basis for all 
obviousness rejections set forth in this Office acdon: 

(a) A patent may not be obtained though the invention is not identically disclosed or described as set forth 
in section 102 of this tide, if the differences between the subject matter sought to be patented and the prior 
art are such that the subject matter as a whole would have been obvious at the time the invention was made 
to a person having ordinary skill in the art to which said subject matter pertains. Patentability shall not be 
negatived by the manner in which the invention was made. 

4. Claims 1, 3, 4 and 8-13 are rejected under 35 U.S.C. 103(a) as being unpatentable 
over Spight (USPN 4,462,046) in view of Corby, Jr. et al. (USPN 5,745,387, hereafter 
Corby). 

5. Regarding claims 1 and 8, Spight discloses a teaching model generating method and 
device for image processing, in which a subject object has the same or substantially similar 
shape as that of a reference object (column 1, line 7-12), the device comprising: an image 
processing system with which current three-dimensional orientation of the subject object 
relative to an image pickup device is recognized based on a plurality of predetermined 
teaching models (stored data of desired object) of the reference object (column 3, line 26-47; 
column 8, line 10-15); and an image-capture system (video detector 40, Figure 1; column 5, 
line 42-48), in advance of the recognizing, generating and storing the plurality of teaching 
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models (reference signals) on the basis of respective image data produced by taking images 
of said reference object from a plurality of directions, so that the image data respectively 
obtained at each of said different image pickup positions, is stored as a teaching model 
(reference signal) (column 9, line 1-19). Spight does not disclose that one of the reference 
object and said image pickup device is fixed to a movable and positionable part of the robot 
or is grasped with a hand of the robot, that said robot is operated for positioning to a 
plurality of different image pickup positions and directions, or that direction information 
indicating the respective different direction is stored with the image data as a teaching model. 
6. Corby discloses an augmented reality maintenance system employing a manipulator 
arm with an archive and comparison device wherein a distal end 100 of the manipulator arm 
10 is attached to a utility package 11, which may include a spatial imaging device such as a 
video camera (Figure 1; column 4, line 18-33). Corby also discloses a position and attitude 
sensing unit 21 to determine the position and orientation distal end 10b, which can then be 
used by manipulator arm renderer 33 to create several images of a prestored model of the 
manipulator arm from model storage 47 from several different viewpoints and environment 
renderer 35 to produce a number of images of the environment corresponding to supplied 
viewpoints (column 4, line 34-65). Corby goes on to disclose an archive and comparison 
device 50 that utilizes a sensor data storage device 51 capable of storing spatial imagery with 
location, orientation and acquisition parameters linked to each image (column 6, line 29-37). 
It would have been obvious to one of ordinary skill in the art at the time the invention was 
made to fix an image pickup device to a movable and positionable part of a robot, operate 
the robot for positioning to a plurality of different image pickup positions and directions, 
and store direction information indicating the respective different direction with the image 
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data as a teaching model as taught by Corby in order to define the identity of the site imaged, 
when it was imaged, the viewpoint, the modality of the imager and description of values 
relating to the image (column 6, line 37-51). 

7. Note also that Spight does not explicidy disclose that the recognized current 
orientation of the subject is three-dimensional (determine object's position and orientation in 
space ; column 8, line 10-15) and Corby only mentions it in passing (column 2, line 6-12; 
column 8, line 41-45). The examiner takes Official Notice that determining three- 
dimensional orientation of an object is well known in the art of robotic vision systems. Lt 
would have been obvious to one of ordinary skill in the art at the time the invention was 
made to determine the three-dimensional orientation of an object in order to determine the 
orientation at which to pick up an object. 

8. Regarding claim 3, Spight discloses that said teaching model is a part of the image 
data of the reference object (column 9, line 10-19; column 10, line 50-63). 

9. Regarding claim 4, Spight discloses that said teaching model comprises data obtained 
by performing image processing on the image data of the reference object (column 9, line 
10-19; column 10, line 50-63). 

10. Regarding claims 9 and 12 (see above discussion of claim 1), Spight discloses a 
method of automatic orientation recognition, comprising: generating and storing a set of 
images of different relative orientations (arrangements; claim 12) of a subject (column 9, line 
1-19), the images having been captured by a plurality of robotic operations corresponding to 
the different relative orientations (arrangements) of the subject (Corby; column 4, line 43- 
65), and associating with each image information indicating its respective relative orientation 
(arrangement) of the subject (Corby column 6, line 29-51); after the generating and storing, 
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from a known current orientation (arrangement) of (an image pickup device on) a robot, 
capturing a current image of a workpiece that has an unknown orientation (arrangement) 
relative to an image pickup device on the robot (Corby; column 4, line 18-33) (before the 
robot has come into contact with the workpiece; claim 9 only), where the workpiece has a 
shape substantially similar to the shape of the subject (Spight; column 3, line 29-38; column 
6, line 11-13); after the capturing, using pattern matching (correlation) to match one of the 
stored images with the current image (column 7, line 28-column 8, line 15; column 9, line 59- 
column 10, line 11); and after the pattern matching, (and before the robot has come into 
contact with the workpiece; claim 9 only), determining the orientation (arrangement) of the 
workpiece relative to the image pickup device on the robot based on the relative orientation 
(arrangement) information associated with the matched stored image (reference signal), and 
also based on the known current orientation (arrangement) of the robot (column 8, line 16- 
37; column 11, line 21-38). 

11. Regarding claim 10, Spight discloses automatically maneuvering the robot to the 
workpiece based at least on the determined orientation of the workpiece relative to the robot 
(column 8, line 16-37). 

12. Regarding claim 11, Spight discloses that the generating and storing is performed for 
a plurality of differently shaped subjects (store a plurality of configurations of each desired 
object; column 9, line 1-19), wherein the current image includes a plurality of differently 
shaped workpieces, and wherein the pattern matching further comprises identifying 
workpieces from among the plurality of differently shaped workpieces using the images and 
orientation information of the plurality of differently shaped subjects (column 6, line 11-13; 
column 7, line 50-column 8, line 15; column 11, line 29-38). 
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13. Regarding claim 13 (see above discussion of claim 1), Spight discloses a method 
comprising: robotically taking images of a subject with different three-dimensional subject- 
camera arrangements that vary in three dimensions (column 9, line 1-19; Corby, column 4, 
line 17-33), and associating with each image or data thereof information indicating its 
subject-camera arrangement (Corby, column 6, line 29-51); then taking a current image of a 
workpiece shaped like the subject (Spight; column 3, line 29-38; column 6, line 11-13); and 
then before picking up the workpiece determining a current workpiece-camera orientation 
by matching (correlating) one of the images or data thereof with the current image (Spight, 
column 7, line 28-column 8, line 15; column 9, line 59-column 10, line 11), and using 
predetermined subject-camera arrangement information of the matched image to determine 
the three-dimensional orientation of the workpiece relative to the camera (Spight, column 8, 
line 16-37; column 11, line 21-38). 

14. Claims 2, 5 and 6 are rejected under 35 U.S.C. 103(a) as being unpatentable over 
Spight in view of Corby in view of Werth et al. (USPN 4,504,970, hereafter Werth). 

15. Regarding claim 2, Spight discloses a teaching model generating device for image 
processing, in which a subject object has same or substantially similar shape as that of a 
reference object, the device comprising: an image processing system with which a current 
three-dimensional orientation of the subject object relative to an image pickup device is 
recognized by carrying out pattern matching processing of an image of the subject based on 
a plurality of pre-determined teaching models of the reference object; and an image capture 
system, in advance of the recognizing, generating and storing the plurality of teaching 
models on the basis of respective image data produced by taking images of said reference 
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object from a plurality of directions, wherein said image pickup device is fixed to a movable 
and positionable part of a robot or is grasped with a hand of the robot, which is operated for 
positioning to a plurality of different relative image pickup positions and directions, so that 
the image data respectively obtained at each of said different image pickup positions is 
stored as a teaching model (see above discussion of claim 1). 

16. Neither Spight nor Corby discloses that the reference object is fixed to a movable 
part of a first robot or is grasped with a hand of the first robot. Werth discloses a training 
controller for pattern processing system wherein it is suggested that an application could 
utilize two robot arms, one which holds a camera which visually guides it to observe a 
precise assembly point and a second which brings a tool or assembly within the visual field 
of the camera where it is visually guided through an operation (column 5, line 12-17). It 
would have been obvious to one of ordinary skill in the art at the time the invention was 
made to utilize two robot arms, one to hold a camera and one to hold a tool or assembly as 
taught by Werth in order to provide more degrees of freedom allowing more views of the 
workpiece from different directions and to provide proper alignment for mating parts in 
automated assembly operations or move a tool to a specific point on the part (column 5, line 
6-11) and more emphasis added, to further enhance the object identification when object is 
viewed at different directions or viewpoints, keeping in view the 3 dimensional structure of 
the object, in real-time and further all the references are directed to robotic industrial 
automation. 

17. Regarding claim 5, Spight discloses that said teaching model is generated for every 
direction in which said image pickup device took the image of said reference object (column 
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9, line 1-19) and Corby discloses that said teaching model is stored in association with the 
information on the direction (column 6, line 29-41). 

18. Regarding claim 6, Spight discloses that said image pickup device 40 is a camera 
(column 5, line 42-48). 

19. Claim 7 is rejected under 35 U.S.C. 103(a) as being unpatentable over Spight in view 
of Corby in view of Werth as applied to claim 2 above, and further in view of Ninomiya et 
al. (USPN 4,611,292, hereafter Ninomiya). 

20. Spight, Corby nor Werth disclose that said image pickup device is a three- 
dimensional visual sensor that measures a distance between the image pickup means and a 
plurality of points on the object. Ninomiya discloses a robot vision system including a three- 
dimensional visual sensor whose image pickup means measures the distance between the 
image pickup means and a plurality of points on the object (column 4, line 28-49). It would 
have been obvious to one of ordinary skill in the art at the time the invention was made to 
utilize a three-dimensional visual sensor as taught by Ninomiya in order to determine the 
position and posture of an object without operation or accuracy being effected by contrast, 
color, or surface condition of the object (column 10, line 35-39). 

Response to Arguments 

21. Applicant's arguments regarding the prior art rejections under Spight and Corby on 
pages 6-9 of the Amendment filed on April 14, 2005, have been fully considered but are not 
persuasive. 
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22. In the 3 rd paragraph of the page 7, 4 th and 5 th paragraph of page 8 and 2 nd and 3 rd 
paragraph of page 9, Applicant argues in substance: 

a. Neither Spight nor Corby. Jr et al.. teach or suggest that "one of the 
reference object and said pickup device is fixed to a movable and positionable part of 

a robot for positioning to a plurality of different iage pickup position and direction 

....[that are] stored as a teaching model" as recited in claim 1. 8. 9 and similarly in 
claims 12 and 13. 

Examiner respectfully disagrees. Spight discloses "the present invention 
relates to use of optical processing for a programmable industrial automation system wherein 
object identification, location, and orientation may be determined" (col. 1, lines 7-12) and 
further discloses robotic vision system which is capable of near real-time sensing and 
analysis, to identify and determine the location and orientation (direction) of workpieces in 
realistic batch manufacturing environment (col. 1, lines 39-44; col. 1, lines 55-60). 

Spight further discloses that in order to produce real-time identification and 
determination of location and orientation of parts (object), a viewed representation of the 
object is compared (pattern matching) with a stored representation of the desired 
objects (apparendy has to be similar shape objects) (col. 1, lines 60-65). From the above 
disclosure by Spight, it is clear that the stored image data is nothing but a plurality of 
teaching models based on the image data of the reference object taken earlier or previously, 
that are required to determine the identity, position and orientation of the object in real- 
time, without which there would be no real time processing and examiner further asserts that 
Spight teaches a robotic vision system with a camera (col. 1, lines 65-68 through lines col. 2, 
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lines 1-5) but Spight does not specifically teach "wherein one of the (either) (a) reference 
object and (or) said image pickup device is fixed to a movable and posirionable part of a 
robot or is grasped with a hand of the robot, and said robot is operated for positioning to a 
plurality of different image pickup positions and directions, so that the image data 
respectively obtained at each of said different image pickup positions and direction 
information indicating the respective different direction, is stored as a teaching model" and 
therefore cites Corby to further provides the support for the teachings missing in Spight. 

Corby clearly teaches a robotic manipulator arm 10 which has a camera installed at 
the movable distal end and which is further used to render the environment which further 
includes the inspection of machines and structures as defined in (col. 1, lines 32-40; col. 4, 
lines 22-34) and inspection of structures as well-known being a part of manufacturing 
process and here is done using the same technique of taking images at different viewpoints 
as described by Spight. Corby clearly teaches "Presendy this is accomplished by moving the 
manipulator arm to a particular position (viewpoint or orientation) and videotaping the 
structure or device (reference object of similar shape) which is to be examined". At a later 
date the manipulator arm is positioned at the same site and current data (such as a video 
image) is compared to previous data" (col. 1, lines 55-65) and it would be apparent to one of 
ordinary skill in the art that if the same structure is to be inspected again, it has to be 
identified before it is examined and Corby clearly teaches moving the camera at a 
viewpoint. Corby further provides teachings of generating images at different orientations or 
directions (viewpoints) (col. 2, lines 33-40; col. 2, lines 45-52), which are further stored and 
used as teaching models based on the image data (col. 4, lines 66-68 through col. 5, lines 1-2; 
col. 6, lines 10-30). Corby further clearly teaches "the sensor device 51 capable of storing 
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spatial imagery with location, orientation and acquisition parameters linked to each image. 
These parameters define the identity of the site imaged (object), when it was imaged, 
the viewpoint, the modality of the imager" (col. 6, lines 35-40). Further, Corby provides 
pattern matching in lines 52-67 of col. 6 through col. 7, lines 1-25, where examiner here 
asserts that pattern matching is nothing but comparing images, to identify the object as 
defined both by Spight and Corby. Therefore, it would have been obvious to one of ordinary 
skill in the art at the time the invention was made to fix an image pickup device to a movable 
and positionable part of a robot, operate the robot for positioning to a plurality of different 
image pickup positions and directions, and store direction information indicating the 
respective different direction with the image data as a teaching model as taught by Corby in 
order to define the identity of the site imaged, when it was imaged, the viewpoint, the 
modality of the imager and description of values relating to the image (column 6, line 37-51) 
and more emphasis added, to further enhance the object identification when object is viewed 
at different directions or viewpoints, keeping in view the 3 dimensional structure of the 
object, in real-time and further both references are directed to robotic industrial automation. 

Arguments regarding claims 2-7 are not persuasive, as applicant argues that 
nothing has been found in Werth or Ninomiya. However, examiner asserts that there being 
no substantial arguments regarding the claims 2-7, these claims still stand rejected under 
same rejections as above. All other arguments regarding claims 1-13 can be additionally 
found in the rejection of claims 1-13 as above. 
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Conclusion 

23. The prior art made of record and not relied upon is considered pertinent to 
applicant's disclosure. 

• Watanabe, U.S. Patent No. 5,329,469, discloses a calibration method for a 
visual sensor. 

• Davis, U.S. Patent No. 6,101,455, discloses automatic calibration of cameras 
and structured light sources. 

• Wu, U.S. Patent No. 5,577,130, discloses a method and apparatus to calculate 
the distance between an image and an object. 

THIS ACTION IS MADE FINAL. Applicant is reminded of the extension of 
time policy as set forth in 37 CFR 1.136(a). 

A shortened statutory period for reply to this final action is set to expire THREE 
MONTHS from the mailing date of this action. In the event a first reply is filed within 
TWO MONTHS of the mailing date of this final action and the advisory action is not mailed 
until after the end of the THREE-MONTH shortened statutory period, then the shortened 
statutory period will expire on the date the advisory action is mailed, and any extension fee 
pursuant to 37 CFR 1.136(a) will be calculated from the mailing date of the advisory action. 
In no event, however, will the statutory period for reply expire later than SIX MONTHS 
from the mailing date of this final action. 
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Any inquiry concerning this communication or earlier communications from the 
examiner should be directed to Manav Seth whose telephone number is (571) 272-7456. 
The examiner can normally be reached on Monday to Friday from 8:30 am to 5:00 pm. 

If attempts to reach the examiner by telephone are unsuccessful, the examiner's 
supervisor, Bhavesh Mehta, can be reached on (571) 272-7453. The fax phone number for 
the organization where this application or proceeding is assigned is 571-273-8300. 

Information regarding the status of an application may be obtained from the Patent 
Application Information Retrieval (PAIR) system. Status information for published 
applications may be obtained from either Private PAIR or Public PAIR. Status information 
for unpublished applications is available through Private PAIR only. For more information 
about the PAIR system, see http://pair-direct.uspto.gov. Should you have questions on 
access to the Private PAIR system, contact the Electronic Business Center (EBC) at 866- 
217-9197 (toll-free). 

Manav Seth 
Art Unit 2625 
August 30, 2005 




